{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization with TensorBoard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualization is a very intuitive way to inspect what is going on in a network. For that purpose, we integrate TensorBoard with MinPy. This tutorial begins with the [Logistic Regression Tutorial](../../get-started/logistic_regression.rst), then moves on to a more advanced case with CNN. \n",
    "\n",
    "Before trying this tutorial, make sure you have installed [MinPy](../../get-started/install.rst) and [TensorFlow](https://www.tensorflow.org/versions/r0.11/get_started/os_setup.html).\n",
    "\n",
    "## Logistic regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up as in the original tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import minpy.numpy as np\n",
    "import minpy.numpy.random as random\n",
    "from minpy.core import grad_and_loss\n",
    "from examples.utils.data_utils import gaussian_cluster_generator as make_data\n",
    "from minpy.context import set_context, gpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up for Visualization. The related code is largely based on the source code from TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from minpy.visualization.writer import LegacySummaryWriter as SummaryWriter\n",
    "import minpy.visualization.summaryOps as summaryOps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Declare the directory for log files which will be used for storing data during the training and later uploaded to TensorBoard later. The directory does not necessarily need to exist. The directory should start with `/private` where TensorBoard looks for log files by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "summaries_dir = '/private/tmp/LR_log'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Predict the class using multinomial logistic regression (softmax regression).\n",
    "def predict(w, x):\n",
    "    a = np.exp(np.dot(x, w))\n",
    "    a_sum = np.sum(a, axis=1, keepdims=True)\n",
    "    prob = a / a_sum\n",
    "    return prob\n",
    "\n",
    "def train_loss(w, x):\n",
    "    prob = predict(w, x)\n",
    "    loss = -np.sum(label * np.log(prob)) / num_samples\n",
    "    return loss\n",
    "\n",
    "\"\"\"Use Minpy's auto-grad to derive a gradient function off loss\"\"\"\n",
    "grad_function = grad_and_loss(train_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the writer for the trainning. You may replace `/train` with `/test`, `/validation`, etc. as you like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_writer = SummaryWriter(summaries_dir + '/train')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`summaryOps.scalarSummary` accepts a tag and a scalar as arguments and crates corresponding summary protos with scalars. `train_writer` then adds the summary proto to the log file. At the end of the training, close the writer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same trick works for all kinds of scalars. To clarify, it works for python scalars `float`, `int`, `long`, as well as one-element `minpy.array.Array` and `numpy.ndarray`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently, Minpy only supports scalar summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Using gradient descent to fit the correct classes.\n",
    "def train(w, x, loops):\n",
    "    for i in range(loops):\n",
    "        dw, loss = grad_function(w, x)\n",
    "        # gradient descent\n",
    "        w -= 0.1 * dw\n",
    "        if i % 10 == 0:\n",
    "            print('Iter {}, training loss {}'.format(i, loss))\n",
    "        summary1 = summaryOps.scalarSummary('loss', loss)\n",
    "        train_writer.add_summary(summary1, i)\n",
    "    train_writer.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize training data.\n",
    "num_samples = 10000\n",
    "num_features = 500\n",
    "num_classes = 5\n",
    "data, label = make_data(num_samples, num_features, num_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 0, training loss [ 14.2357111]\n",
      "Iter 10, training loss [ 1.60548949]\n",
      "Iter 20, training loss [ 0.25217342]\n",
      "Iter 30, training loss [ 0.10623146]\n",
      "Iter 40, training loss [ 0.06312769]\n",
      "Iter 50, training loss [ 0.0435785]\n",
      "Iter 60, training loss [ 0.03269563]\n",
      "Iter 70, training loss [ 0.02586649]\n",
      "Iter 80, training loss [ 0.02122972]\n",
      "Iter 90, training loss [ 0.01790031]\n"
     ]
    }
   ],
   "source": [
    "# Initialize training weight and train\n",
    "weight = random.randn(num_features, num_classes)\n",
    "train(weight, data, 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the terminal, and call the following command:\n",
    "\n",
    "~~~bash\n",
    "tensorboard --logdir=summaries_dir\n",
    "~~~\n",
    "\n",
    "Note you don't need to include `/private` for the `summaries_dir`, so in this case the `summaries_dir` will be `/tmp/LR_log`.\n",
    "\n",
    "Once you start TensorBoard, you should see the visualization of scalars in the EVENTS section as below. When you move your mouse along the curve, you should see the value at each step. You may change the size of the graph by clicking the button in the bottom-left corner.\n",
    "\n",
    "![Loss History Interface](loss.png)\n",
    "\n",
    "On the left hand side, you may decide to which extent you want to smooth the curve. You may choose one of the three choices as the horizontal axis of the graph. By checking **Data download links**, you may download the data in the format of `.csv` or `.json`. In the `.csv` or `.json` files, the data will be displayed in the form of [wall time - step - value].\n",
    "\n",
    "![Control Interface](control.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
